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To  Ask  a Question,  One  Must  Know  Enough  to  Know  What  is  Not  Known 
Naomi  Miyake  and  Donald  A.  Norman 


To  ask  a question  of  someone  implies  more  than  a need  for  informa- 
tion. It  also  implies  a proper  structure  of  knowledge  with  which  to 
formulate  the  question  and  to  interpret  the  response.  Thus,  the  ability 
of  a person  to  think  of  an  appropriate  question  on  a topic  matter  is  a 
complex  function  of  the  knowledge  of  that  topic. 

At  a research  seminar  on  computer  techniques,  we  noted  that 
beginners  at  programming  (to  whom  the  seminar  was  addressed)  asked  few 
questions  and  generated  few  comments.  More  expert  programmers,  however, 
had  many  questions  and,  eventually,  dominated  the  discussion.  Reflec- 
tion on  this  observation  indicated  that  a general  principle  might  be 
involved:  in  order  to  be  able  to  ask  questions  about  new  material,  one's 
knowledge  of  the  topic  must  be  at  an  appropriate  level:  the  number  of 
questions  should  be  an  inverted  U-shaped  function  of  the  relationship 
between  student  knowledge  and  task  complexity. 

Consider  the  novice  exposed  to  expert  level  material:  although 
there  should  be  many  difficulties  in  understanding,  the  novice  does  not 
even  have  the  proper  framework  within  which  to  ask  questions.  Thus, 
although  at  first  glance,  one  might  expect  novices  to  be  filled  with 
questions,  there  are  strong  theoretical  reasons  not  to  expect  this. 

Consider  the  expert  exposed  to  novice  material.  Here,  there  should 
be  good  structural  framework  for  the  material  on  the  part  of  the  expert, 
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but  a lack  of  questions:  there  are  no  gaps  or  inconsistencies  in  the 
knowledge  that  need  to  be  filled.  The  maximum  number  of  questions 
should  occur  when  a person  is  well  matched  to  the  level  of  presentation. 
There  should  be  sufficient  background  knowledge,  but  the  new  material 
should  indeed  be  new,  with  inconsistencies  or  incomplete  information 
that  causes  the  learner  to  question  the  interpretation,  to  seek  more 
information,  or  to  confirm  the  existing  'understanding. 

The  ability  to  ask  questions  during  the  learning  of  new  material, 
therefore,  seems  to  be  a function  of  two  variables:  the  existence  of 
appropriate  knowledge  structures,  and  the  level  of  completeness  of  those 
structures  with  respect  to  the  new  material.  When  there  are  two  presen- 
tation levels  of  the  same  material,  easy  and  hard,  the  learner  with  the 
ample  knowledge  structure  should  ask  more  questions  on  the  harder 
presentation  and  less  on  the  easier  one;  the  learner  with  little 
knowledge  should  ask  most  questions  with  the  easier  presentation  and  not 
many  with  the  harder  presentation. 

Considerable  research  has  been  done  on  question  answering,  but  we 
have  been  unable  to  find  relevant  experimental  or  theoretical  studies  on 
the  generation  of  questions.  Some  relevant  Issues  are  discussed  in  the 
speech-act  literature,  but  the  relevance  is  indirect.  Grosz  (Note  1) 
and  Levin  and  Moore  (1977)  have  looked  at  a naturalistic  dialogue  situa- 
tions, but  they  did  not  perform  any  analyses  relevant  to  the  issues 
being  raised  here. 

This  paper  tests  the  predicted  interaction.  The  topic  to  be 
learned  is  a computer  text  editor.  This  is  a topic  for  which  it  is  easy 


3 


to  find  subjects  with  no  prior  knowledge.  Thus,  it  is  relatively  easy 
to  manipulate  the  amount  of  prior  knowledge,  as  well  as  the  difficulty 
level  of  the  material. 

Method 

Design 

We  used  two  levels  of  background  knowledge  and  two  levels  of  diffi- 
culty of  the  material  to  be  learned.  Subjects  were  preselected  to  have 
little  or  no  prior  knowledge  about  text  editors  or  computers.  Trained 
subjects  received  a pre-experimental  training  session.  Other  subjects 
(novices)  did  not.  There  were  two  levels  of  the  learning  task:  the 
level  called  Easy  was  quite  less  advanced  than  the  level  called  Hard. 
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There  were  four  groups  of  subjects,  with  15  subjects  per  group: 
trained,  Easy;  trained,  Hard;  novice,  Easy;  novice,  Hard.  The  order  of 
the  conditions  were: 

1.  training  (for  subjects  in  trained  conditions  only); 

2.  a practice  task; 

3.  a baseline  task; 

4.  the  learning  task. 

Training 

During  the  training  session,  subjects  learned  a task  related  to  the 
final  learning  task.  Using  a simplified  training  manual,  the  subjects 
in  the  two  "trained"  groups  learned  three  commands  of  the  text  editor 
called  ED:  how  to  print,  how  to  delete,  and  how  to  add  a line  of  text. 
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After  practicing  these  three  commands  on  the  computer  terminal,  the  sub- 
jects were  given  three  new  texts,  each  of  which  had  mistakes  marked: 
they  were  asked  to  fix  the  mistakes.  They  could  refer  back  to  the 
training  manual.  During  the  training,  the  experimenter  was  available 
for  help,  but  no  more  Information  than  that  given  in  the  training  manual 
was  provided.  The  amount  of  training  was  selected  to  be  sufficient  for 
subjects  to  gain  a sense  of  the  text  editor.  The  training  session  took 
some  25  to  50  minutes,  depending  on  the  ability  of  each  subject.  All 
the  trained  subjects  reached  the  level  where  they  could  fix  at  least  one 
text  out  of  three  without  any  help. 

Ihfi.  Practice  .and.  baseline  tasks 

The  goal  of  the  experiment  was  to  collect  freely  occurring  thoughts 
and  questions  of  the  subjects  who  were  learning  different  levels  of 
materials.  Accordingly,  it  was  felt  important  to  train  the  subjects  in 
giving  protocols  naturally.  (It  takes  a little  bit  of  training  to  speak 
thoughts  and  questions  aloud  freely,  especially  when  the  thoughts  are 
not  responded  to.)  In  addition,  because  different  subjects  would  have 
different  baseline  rates  of  asking  questions  and  speaking,  we  wished  to 
get  a calibration  of  their  baseline  rate. 

All  the  materials  for  the  practice,  baseline,  and  learning  tasks 
were  presented  on  individual  index  cards,  held  in  a loose-leaf  binder. 
Each  card  had  one  or  two  sentences.  Subjects  read  each  card  and  were 
instructed  to  say  their  thoughts  and  questions  aloud. 


The  practice  material  was  three  sentences  taken  from  an  article  in 
a popular  science  magazine  (Mahoney,  1978).  Each  sentence  was  on  a 
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separate  index  card.  After  each  subject  finished  reading  and  responding 
to  the  third  sentence,  all  the  quesquestions  up  to  that  point  were 
answered  by  the  experimenter.  There  was  only  one  subject  who  did  not 
say  anything,  and  so  he  was  given  the  two  most  common  questions  and 
their  answers  as  examples. 

The  baseline  task  was  to  learn  to  play  Owari,  an  African  board  and 
counter  game,  quite  different  from  most  board  games  in  the  United 
States.  The  explanation  was  made  deliberately  obscure.  It  consisted  of 
six  very  short  paragraphs  (only  one  or  two  sentences  each),  each  on  a 
separate  index  card.  Subjects'  questions  were  again  answered  at  the  end 
of  this  session.  (All  subjects  asked  questions  in  the  baseline  task.) 

Ihs.  learning  laak 


Two  versions  of  the  instructional  manual  for  the  text  editor  NED 
were  constructed.  The  Easy  Manual  was  readable  for  people  with  no 
experience  with  computers;  it  used  only  non-technical  language,  aave 
detailed  explanation  for  concepts,  and  taught  five  commands  by  first 
presenting  concrete  examples  and  later  more  abstract  explanations.  The 
Hard  Manual  was  more  technical,  written  for  experienced  readers.  It 
taught  seven  commands  and  three  special  characters.  It  presented 
abstract  conceptual  explanations  first,  followed  by  a specification  of 
how  to  do  each  command.  Both  the  Easy  and  Hard  Manuals  were  divided 
into  30  steps,  each  of  which  consisted  of  one  or  two  sentences,  each 
step  on  a separate  index  card.  (NED  differs  from  ED,  the  editor  used  in 
the  training.  The  instruction  manual  used  for  ED  was  considerably  dif- 
ferent from  either  of  the  manuals  used  in  the  learning  task  for  NED.) 
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During  the  learning  taste,  subjects  were  asked  to  read  the  material 
page  by  page  and  to  say  aloud  all  the  questions  and  thoughts  which  came 
to  mind.  The  instructions  were  specifically  aimed  at  making  the  task 
natural.  All  the  utterances  were  tape  recorded.  In  order  to  keep  the 
amount  of  information  available  the  same  from  subject  to  subject,  the 
experimenter  never  responded  to  these  questions. 

Subjects 

Sixty  undergraduates  in  introductory  psychology  courses  at  the 
University  of  California,  San  Diego  served  as  subjects,  either  for 
course  credits  or  pay.  They  were  pre-selected  to  have  no  knowledge  of 
either  text  editors  or  computers.  It  was  hoped  that  no  subjects  would 
be  familiar  with  the  base  line  task,  Owari.  However,  12  subjects  recog- 
nized the  game  while  reading  the  explanation.  They  were  disqualified  and 
replaced  with  new  subjects. 

ReaulU 

Cflfllaft  Transcripts 

All  the  protocols  recorded  during  the  baseline  and  learning  tasks 
were  transcribed.  There  does  not  yet  exist  an  adequate  method  for 
analyzing  naturally  occuring  questions.  Considerable  syntactic  and 
semantic  variations  were  observed  in  these  studies.  Accordingly,  we 
first  devised  a rough  classification  of  the  utterances  into  ten  dif- 
ferent categories.  It  was  difficult  to  make  unambiguous  classifications 
of  all  the  questions  into  these  ten  categories,  however,  and  so  these 
categories  were  grouped  into  four  composite,  more  conceptual  categories 


(plus  a fifth  category,  looking  back).  Four  of  the  transcripts,  one 
from  each  experimental  group,  were  randomly  selected  and  coded  by  an 
independent  Judge  and  the  results  were  compared  with  the  codings  by  one 
of  the  authors  (N.M.).  Between  coder  reliability  was  only  about  0.7  on 
the  ten  categories,  but  was  0.95  on  the  composite  classifications. 
Examples  of  the  questions  and  the  categorizations  used  are  shown  in 
Table  1.  Only  the  five  categories  are  considered  in  this  paper.  They 
are: 

1.  Overall  questions:  The  total  number  of  questions,  including 
repetitions,  paraphrases  of  previously  asked  questions,  and  margi- 
nal  questions  (an  interrogative,  but  with  no  substantial  content). 

2.  Number  of  concepts:  The  number  of  concepts  for  which  further 
clarification  was  asked.  Operationally,  this  i3  the  total  number 
of  questions,  minus  repetitions,  paraphrases  and  marginal  ques- 
tions. 

3.  Hypotheses:  The  number  of  specific  hypotheses  or  subjects'  own 
interpretations--a  measure  of  how  easily  subjects  could  speculate 
about  the  possible  answers  to  their  own  questions. 

The  composite  categories  "number  of  concepts"  and  "hypotheses" 
are  both  subsets  of  the  composite  category  "overall  questions": 
moreover,  the  two  overlap  in  sharing  some  common  subcategories. 

4.  Confirmations:  The  number  of  utterances  (other  than  questions) 
in  which  subjects  stated  their  own  interpretation  of  the  material. 
Mere  reading  aloud  was  not  included.  This  category  reflects  sub- 
jects' understanding  of  the  material,  but  with  an  apparent  need  for 
confirmation  of  the  interpretation. 
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Table  1 

Coding  Categories,  Example  statements,  and  Composite  Categories. 
(In  parentheses  are  shown  previously  asked  questions.) 


Categories 

Example  statements 

Questions 

With  contents 

Original 

With  hypotheses 

"Is  NED  a person?" 

Without  hypotheses 

"What  does  NED  stand  for?" 

Paraphrased 

With  hypotheses 

"(Is  NED  a person?)  Or  a computer?" 

Without  hypotheses 

"(What  is  the  comnnad?)  What  type  of 

thing  do  you  want  to  type  in,  that 

would  be  considered  as  command?" 

Repeated 

"(What  is  the  buffer?)  Again,  what  is 

the  buffer?" 

Without  contents 

"What.. .?" 

Judgements 

"I  am  confused." 

Confirmations 

"Oh,  so  insert  goes  backward,  instead 

of  forward  like  append  does." 

Verbatim  reading  aloud 

Composite  Categories: 

Overall  questions  = All  statements  categorized  as  "Questions"  above. 

Humber  of  concepts  = All  statements  categorized  as  "Questions  - 

With  contents  - Original"  above. 

Hypotheses  = "Questions  - With  contents  - Original  - With  hypotheses" 

♦ "Questions  - With  contents  - Paraphrased  - With  hypotheses." 
Confirmations  = All  statements  categorized  as  "Confirmations"  above. 

= J 
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5.  Look-backs:  The  total  number  of  the  times  subjects  turned  back 
to  previous  pages.  The  number  of  pages  turned  back  was  not 
counted.  (The  data  were  recorded  by  the  experimenter  during  the 
baseline  and  learning  tasks.) 

Adjustment  lor  tendency  1a  flUaaUgfla 

There  were  considerable  individual  differences  in  the  tendency  to 
speak  aloud.  To  compensate  for  this  propensity  to  speak,  the  number  of 
utterances  in  the  experimental  condition  was  adjusted  by  using  the 
number  of  questions  observed  in  the  baseline  condition.  The  argument 
for  this  adjustment  is  simple.  In. the  baseline  condition,  all  subjects 
were  novices.  Therefore,  differences  in  the  number  of  questions  could 
only  result  from  individual  differences  in  speaking  styles.  Moreover, 
there  were  no  significant  differences  in  mean  baseline  frequency  among 
the  four  experimental  groups.  The  correlation  between  overall  questions 
in  the  baseline  condition  and  overall  questions  in  the  learning  task  was 
.66.  The  regression  equation  for  the  number  of  overall  questions  was: 

TQ  = 6.2  + 1.7  BQ, 

where  TQ  is  the  learning  task  questions  and  BQ  is  the  baseline  ques- 
tions. Accordingly,  the  slope  of  the  regression  equation  was  used  to 
determine  the  adjustment  factor:  the  number  of  overall  questions,  number 
of  concepts,  and  hypotheses  were  divided  by  1.7BQ.  All  statistical  ana- 
lyses were  done  on  both  raw  scores  and  adjusted  scores,  and  both  values 
will  be  reported.  (We  used  the  slope  of  the  regression  equation  for  our 

The  mean  number  of  overall  baseline  questions  was  4.7  for  the  novice 
subjects  with  the  Easy  Manual,  4.9  for  the  trained  subjects  with  the 
Easy  Manual,  7.2  for  the  novice  subjects  with  the  Hard  Manual,  5.5  for 
the  trained  subjects  with  the  Hard  Manual. 
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correction  because  it  provided  a satisfactory  and  easy  estimate.  Note, 
however,  that  our  results  are  not  sensitive  to  the  form  of  the  adjust- 
ment used.  The  primary  point  of  the  paper  is  confirmed  by  using  the  raw 
data,  without  adjustment.  Performing  the  adjustment  does  allow  some 
secondary  results  to  reach  significance. ) 

The  number  of  "confirmations"  and  "look-backs"  were  not  adjusted 
because  the  baseline  frequencies  for  more  than  40  subjects  were  zero 
(this  is  about  2/3  of  the  subjects).  Moreover,  the  correlation  between 
the  baseline  and  the  learning  phase  for  these  two  categories  was  low  (r 
= .23  and  .38,  for  confirmations  and  look-backs  respectively). 

findings 

The  results  for  questions  confirm  expectations.  All  the  adjusted 
questioning  scores  show  significant  interactions  between  the  training 
conditions  and  the  levels  of  the  instructional  manuals:  with  the  easier 
manual,  novice  subjects  asked  more  questions  than  trained;  with  the 
harder  manual,  trained  subjects  asked  more  questions  than  novices.  Fig- 
ure 1 shows  the  pattern  based  on  the  number  of  concepts,  the  most  rigid, 
qualitative  measure  of  questioning.  The  £ values  for  the  interactions 
calculated  from  the  2X2  analysis  of  variance  (training  conditions  by 
manual  levels)  are,  £ (1,  56)  = 32.34,  £ < *01  for  overall  questions;  £ 
(1,  56)  =. 32.35,  £ < .01  for  the  number  of  concepts;  £ (1,  56)  = 6.59,  £ 
< .05  for  hypotheses.  1 

1.  Main  effects  were  not  significant,  except  for  training  conditions  on 
overall  questions  (£  (1,  56)  = 6.93,  £ < *05)  and  on  concepts  (£  ( 1 , 56) 
= 4.47,  £ < .05).  Main  effects  themselves  do  not  convey  much  informa- 
tion, because  the  experimental  hypotheses  clearly  call  for  the  signifi- 
cant interactions. 
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The  seen  questioning  scores  (both  raw  and  adjusted)  are  shown  in 
Table  2.  With  the  exception  of  the  hypotheses,  all  the  raw  frequencies 
show  the  interacting  pattern  in  the  predicted  order.  The  £ values  for 
the  interactions  are  £ (1,  56)  = 12.39,  Ji  < .01  for  overall  questions;  £ 
(1,  56)  s 11.72,  s.  < .01  for  the  number  of  concepts;  £ (1,  56)  = 1.90, 
for  hypotheses.  2 

Simple  main  effects  for  the  factor  of  manual  levels  are  mostly  sig- 
nificant. All  the  questioning  scores  on  the  Easy  Manual  are  signifi- 
cantly greater  for  novices  than  for  trained  subjects.  The  £ values  are 
£ (1,  56)  a 34.81,  £ < .01  for  overall  questions;  £ (1,  56)  a 30.50,  a < 
.01  for  concepts;  £ (1,  56)  = 13.76,  £ < .01  for  hypotheses.  On  the 
Hard  Manual,  where  the  question  is  whether  the  trained  subjects  asked 
more  questions  than  the  novices,  only  the  category  of  hypotheses  fails 
to  reach  the  significant  level  difference.  The  £ values  for  overall 
questions,  concepts,  and  hypotheses  are  £ (1,  56)  = 4.70,  £ < .05;  £ (1, 
56)  s 6.39,  jj,  < .05;  and  £ (1,  56)  = 0.45,  respectively. 

Mean  number  of  confirmations  and  look-backs  are  shown  in  Fig-ore  2 
and  Table  3 (these  are  raw,  unadjusted  data).  Although  the  interaction 
of  confirmation  does  not  reach  significance,  (£  (1,  56)  a 2.22),  the 
data  do  show  the  appropriate  pattern  of  interaction.  Look-backs  have  a 
unique  pattern,  with  neither  significant  interaction  (£  (1,  56)  a 0.30), 
nor  any  significant  differences  between  the  novices  and  the  trained  sub- 
jects. The  Hard  Manual  elicits  more  look-backs  than  the  Easy  Manual 
(main  £ for  manual  levels  is  £ (1,  56)  a 4.92,  n.  < .05). 


2.  Only  one  main  effect,  the  training  condition  factor  on  overall  ques- 
tions (£  (1,  56)  a 5.44,  SL  < .05)  is  significant. 
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Table  2 

Mean  Number  of  Questions  Asked 


While  Learning  the  Easy  Manual  or  the  Hard  Manual . 
(Standard  deviations  are  given  in  parentheses.) 


Easy  Manual 

Hard 

Manual 

Novice 

Trained 

Novice 

Trained 

Raw  frequencies 

Overall 

20.9 

6.5 

16.2 

19.1 

(11.9) 

( 4.5) 

(10.1) 

(10.1) 

Concepts 

15.1 

5.9 

12.5 

15.3 

( 7.3) 

( 3.9) 

( 7.6) 

( 7.6) 

Hypotheses 

7.0 

1.5 

7.5 

7.2 

( 7.5) 

( 1.7) 

( 9.D 

( 8.5) 

2.2 

( 0.9) 

1.8 

( 0.7) 

0.7 

( 0.8) 


Adjusted  scores 


Overall 

2.8 

0.9 

1.5 

( 1.2) 

( 0.7) 

( 0.5) 

Concepts 

2.1 

0.8 

1.1 

( 0.8) 

( 0.6) 

( 0.4) 

Hypotheses 

0.9 

0.2 

0.6 

( 0.9) 

( 0.2) 

( 0.3) 
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Table  3 

Mean  Number  of  Confirmations  and  Look- backs. 
(Standard  deviations  are  given  in  parentheses.) 


Easy  Manual 

Hard 

Manual 

Novice 

Trained 

Novice 

Trained 

Confirmations 

2.5 

1.3 

1.0 

2.9 

( 4.5) 

( 2.4) 

( 2.1) 

( 6.0) 

Look- backs 

1.3 

1.7 

2.7 

3.9 

( 1.3) 

( 2.2) 

( 1.9) 

( 5.3) 

Piacuaaloa 


The  most  interesting  result  is  that  novices  do  not  ask  many  ques- 
tions on  material  that  is  too  difficult.  This  finding  shows  that  a 
theory  of  questioning  that  suggests  that  people  ask  questions  to  fill  in 
their  knowledge  structures  is  too  simplistic.  People  do  not  appear  to 
be  able  to  cope  with  material  too  far  beyond  their  present  knowledge. 

The  number  of  overall  questions  gives  a quantitative  measure  of  the 
general  tendency  to  ask  questions.  The  number  of  concepts  and 
hypotheses  can  be  viewed  as  reflecting  more  qualitative  aspects  of  ques- 
tioning. To  ask  about  a concept,  the  subjects  must  know  what  is  missing 
and  what  is  necessary  for  . farther  understanding;  questions  asking  about 
concepts  thus  imply  the  asker  is  at  an  appropriate  distance  from  the 
given  material.  To  create  a hypothesis,  subjects  should  have  some 
expectation  or  inferred  understanding  beyond  the  given  material; 
hypotheses  thus  imply  the  asker  is  active  in  constructing  a knowledge 
structure.  In  this  sense,  the  novices  who  learned  on  the  easier  manual 
and  the  trained  subjects  on  the  harder  manual  had  the  greater  tendency 
to  ask  questions,  not  only  quantitatively  but  qualitatively. 

Because  the  two  Instructional  manuals  differed  in  structural  pro- 
perties, it  is  possible  to  infer  what  subjects  in  each  group  were  doing. 
The  Easy  Manual  first  presented  concrete  examples  and  later,  more 
abstract  explanations;  the  Hard  Manual  presented  the  abstract  concepts 
first  and  then,  later,  the  detailed  specification  of  how  to  use  each 


command 
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The  novices  given  the  Easy  Manual  started  without  knowledge  of  the 
text  editor  task,  but  the  manual  provided  concrete  examples  which  could 
be  used  as  the  "core"  to  start  learning.  Thus,  these  subjects  could 
follow  the  manual,  constantly  expanding  the  scope  of  their  knowledge. 

The  novices  on  the  Hard  Manual  also  started  without  any  specific 
knowledge,  but  had  to  cope  immediately  with  the  abstract  framework.  It 
was  as  if  they  did  not  know  what  they  should  know  to  know  further.  3y 
the  time  they  came  to  more  concrete  examples,  it  was  too  late. 

Look-backs  can  be  regarded  either  as  a measure  of  the  complexity  of 
the  material  or  as  a measure  of  memory  loss.  There  should  be  more 
memory  loss  for  the  novices  on  the  Hard  Manual,  because  they  had  the 
greatest  trouble  in  following  the  material.  Similarly,  trained  subjects 
on  the  Hard  Manual  should  need  to  check  back  on  the  materials,  because 
they  were  taking  in  the  largest  amount  of  information.  These  two  types 
of  look-backs  are  indistinguishable  with  the  present  data.  The  data  for 
look-backs  are  consistent  with  either  interpretation.  However,  the 
relatively  high  score  of  the  novices  with  the  Hard  Manual  implies  that 
they  were  actually  trying  to  understand  the  material,  not  Just  flipping 
the  page  with  boredom;  they  simply  did  not  know  what  they  should  ask. 

This  paper  is  one  of  the  first  experimental  attempts  to  manipulate 
question  asking  in  a learning  task.  The  results  show  that  at  least  some 
aspects  of  complex  learning  are  accessible  through  the  measures  of  ques- 
tion asking.  The  results  also  show  that  any  theory  of  question  asking 
in  learning  cannot  simply  use  the  gaps  in  a person's  knowledge  as  the 
source  of  questions.  To  ask  a question,  knowing  too  little  is  Just  as 
disadvantageous  as  knowing  too  much. 


I 
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